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(57) Abstract 

A sports event video manipulating system (10) for manipulating a representation of a sports event, the sports editor including a video 
field grabber (40) operative to grab at least one video field including a video image, an A/D converter (20) operative to digitize a grabbed 
video field, an object tracker (30, 50) operative to track an object through a plurality of successive video fields, an object highlighter (30, 
50) receiving input from the object tracker (30, 50) and operative to highlight the tracked object on each of the plurality of successive video 
fields, a D/A image converter (20) operative to convert output of the object highlighter (30, 50) into a video standard format, and a video 
display monitor (70). 
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1 SPORTS EVENT VIDEO 

2 

3 FIELD OF THE INVENTION 

4 

5 The present invention relates to editing and 

6 processing of video segments. 
7 

8 
9 

10 BACKGROUND OF THE INVENTION 

11 

12 . Documentation, of team games such as basketball, 

13 football and soccer on video is useful for coaches and 

14 players who view the video film in order to understand 

15 their own and opponents* past performance. Sports com- 

16 mentators also view and show video representations of 

17 team games in the course of analyzing these games for 

18 their viewers* For this purpose, commercially available 

19 analog video editors are currently employed, such as 

20 Sports-Tech Video Editor (STVE) of Sports-Tech. Inc. of 

21 Fort Lauderdale, Florida, 

22 - 
23 

24 

25 SUMMARY OF THE INVENTION 

26 

27 

28 The present invention seeks to provide a digi- 

29 tal sports event video manipulating system which allows a 

30 user to . manipulate a video representation of a sport 

31 event by deriving digital information regarding the 

32 sports event from the video representation and manipulat- 

33 ing the digital information. 

34 There is thus provided in accordance with a 

35 preferred embodiment of the present invention a sports 

36 event video manipulating system for manipulating a repre- 
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1 sentation of a sports event, the sports editor including 

2 a video field grabber operative to grab at least one 

3 video field including a video image A/D converter opera- 

4 tive to digitize a grabbed video field, an object tracker 

5 operative to track an object through a plurality of 

6 successive video fields, an object highlighter receiving 

7 input from the object tracker and operative to highlight 

8 the tracked object on each of the plurality of successive 

9 video fields, a D/A image converter operative to convert 

10 output of the object highlighter into a video standard 

11 format, and a video display monitor. 

12 Further in accordance with a preferred erabodi- 

13 ment of the present invention, the video field grabber 

14 is replaced by a video frame grabber. 

15 Still further in accordance with a preferred 

16 embodiment of the present invention, the system also 

17 includes a marking device allowing a user to indicate an 

18 object of interest and providing an output indication of 

19 the object of interest to the object tracker. 

20 Still further in accordance with a preferred 

21 embodiment of the present invention, the system includes 

22 a video editor operatively associated with the video 

23 field grabber and with the video display monitor. 

24 Additionally in accordance with a preferred 

25 embodiment of the present invention, the object tracker 

26 includes an object exiting monitor operative to monitor 

27 for exit of an object from at least one video field. 

28 Further in accordance with a preferred embodi- 

29 ment of the present invention, the object tracker is 

30 operative to track a plurality of objects. 

31 Still further in accordance with a preferred 

32 embodiment of the present invention, the object tracker 

33 includes an occlusion predictor operative to predict 

34 occlusion of at least one of the plurality of tracked 

35 objects by at least one other object. 

36 Additionally in accordance with a preferred 
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1 embodiment of the present invention, the object tracker 

2 includes a moving object identifier operative to identify 

3 objects which are in motion at least some of the time as 

4 the plurality of objects to be tracked. 

5 Still further in accordance with a preferred 

6 embodiment of the present invention, the moving object 

7 identifier includes a color analyzer operative to. distin- 

8 guish at least one color characterizing objects at least 

9 sometimes in motion from at least one background color 

10 characterizing stationary objects. 

11 Additionally in accordance with a preferred 

12 embodiment of the present invention, the object tracker 

13 includes actual location computation means operative to 

14 compute and store an indication of an actual location of 

15 a tracked object at an individual time. 

16 Further in accordance with a preferred erabodi- 

17 ment of the present invention, the object tracker in- 

18 eludes f ield-of-view direction determining means opera- 

19 tive to provide an indication of the actual direction of 

20 a current field of view center. 

21 There is also provided, in accordance with 

22 another preferred embodiment of the present invention, a 

23 video imagery manipulating system for manipulating video 

24 imagery including a video field grabber operative to grab 

25 at least one video field, an object tracker operative to 

26 track an object through a plurality of successive video 

27 fields, and an object highlighter receiving input from 

28 the object tracker and operative to highlight the tracked 

29 object on. each of the plurality of successive video 

30 fields. 

31 — There is .also provided, in accordance with 

32 another preferred embodiment of the present invention, a 

33 sports event video manipulating system for manipulating a 

34 representation of a sports event, the sports editor 

35 including video field grabber operative to grab at least 

36 one video field including a video image A/D converter 
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1 operative to digitize a grabbed video field, and field 

2 mosaic composer operative to receive a plurality of 

3 digitized fields from the video field grabber, represent- 

4 ing a corresponding plurality of small portions of an 

5 arena and to compose a mosaic of fields representing a 

6 larger portion of the arena. 

7 Further in accordance with a preferred embodi- 

8 ment of the present invention, the system also includes 

9 a field mosaic graphic output system operative to pss>- 

10 vide a visually sensible representation of the larger 

11 portion of the arena. 

12 Still further in accordance with a preferred 

13 embodiment of the present invention, the field mosaic 

14 graphic output system includes a video display, and a D/A 

15 converter operative to convert a digital representation 

16 of the field mosaic to a video representation thereof. 



17 Additionally in accordance with a preferred 

18 embodiment of . the present invention, the field mosaic 

19 graphic output system includes a computer screen. 

20 Further in accordance with a preferred embodi- 

21 ment of the present invention, the field mosaic graphic 

22 output system includes a printer. 

23 Still further in accordance with a preferred 



24 embodiment of the present invention, the video field 

25 grabber includes a video frame grabber operative to grab 

26 at least one video frame and wherein the A/D converter is 
.27 operative to digitize a grabbed video frame and wherein 

28 the field mosaic composer includes a frame mosaic compos- 

29 er operative to receive a plurality of digitized frames 

30 from the video frame grabber and to compose therefrom a 

31 mosaic of frames. 

32 Still further in accordance with a preferred 

33 embodiment of the present invention, the plurality of 

34 small portions are viewed from a corresponding, plurality 

35 of distances, the differences between at least some of 

36 the distances being comparable in size to the distances 
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1 themselves. 

2 Further in accordance with a preferred erabodi- 

3 ment of the present invention, the plurality of small 

4 portions are viewed from a single location. 

5 There is also provided, in accordance with 

6 another preferred embodiment of the present invention, a 

7 sports event video manipulating system for manipulating a 

8 representation of a sports event, the sports editor 

9 including a video field grabber operative to grab a video 

10 field sequence including a video image A/D converter 

11 operative to digitize a grabbed video field, and a 

12 motion-based foreground-background discriminator opera- 

13 tive to differentiate foreground objects in the video 

14 field sequence from background in the video field se- 

15 quence, at least partly on the basis of foreground object 

16 motion. 

17 Further in accordance with a preferred embodi- 

18 ment of the present invention, the system includes a 

19 field mosaic composer operative to receive a sequence of 

20 pluralities of digitized fields from the video field 

21 grabber, each plurality of digitized fields representing 

22 a corresponding plurality of small portions of an arena 

23 and to compose a sequence of mosaics of fields, each 

24 mosaic representing a larger portion of the arena and the 

25 discriminator is operative to differentiate foreground 

26 objects in the sequence of mosaics. 

27 There is also provided, in. accordance . with a 

28 further preferred embodiment of the present invention, a 

29 sports event video manipulating system for manipulating a 

30 representation of a sports event, the sports editor 

31 including a video field grabber operative to grab at 

32 least one video field. including a video image A/D con- 

33 verter operative to digitize a grabbed video field, and 

34 a foreground object shape f oreground-background. discrimi- 

35 nator operative to differentiate foreground objects in 

36 the video field from background in the video field, at 
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least partly on the basis of foreground object shape. 

There is additionally provided, in accordance 



3 with another preferred embodiment of the present inven- 

4 tion, a sports event video manipulating system for 

5 manipulating a representation of a sports event, the 

6 sports editor including a video field grabber operative 

7 to grab at least one video field including a video image 

8 A/D converter operative to digitize a grabbed video 

9 field, and a foreground object shape foreground-back- 

10 ground discriminator operative to differentiate a charac- 

11 ter-bearing foreground object in the video field from 

12 background in the video field, at least partly on the 

13 basis of character recognition. 

14 Further in accordance with a preferred embodi- 

15 ment of the present invention, the discriminator differ- 

16 entiates foreground objects from background objects at 

17 least partly on the basis of color. 

18 Still further in accordance with a preferred 

19 embodiment of the present invention, the system includes 

20 a foreground object eliminator operative to eliminate 

21 foreground objects and replace them with adjacent back- 

22 ground information. 

23 Further in accordance with a preferred embodi- 

24 ment of the present invention, the system also includes a 

25 selected object replacer operative to identify a selected 

26 object and to replace the selected object with an icon in 

27 the mosaic. 

28 Still further in accordance with a preferred 

29 embodiment of the present invention, the video indexing 

30 method includes the steps of providing a digital repre- 

31 sentation of a video sequence featuring at least one 

32 object performing at least one type of action, and index- 

33 ing the video sequence according to at least one index. 

34 Further in accordance with a preferred embodi- 

35 ment of the present invention, the step of indexing 

36 includes the step of indexing according to the identity 
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1 of the object. 

2 Still further in accordance with a preferred 

3 embodiment of the present invention, the step of indexing 

4 includes the step of indexing by action type. 

5 Additionally in accordance with a preferred 

6 embodiment of the present invention, the video sequence 

7 represents a sports event and the step of indexing ac- 

8 cording to the identity of the object includes the step 

9 of indexing according to the identity of at least one 

10 player participating in at least a portion of the sports 

11 event. 

12 Still further in accordance with a preferred 

13 embodiment of the present invention, the step of indexing 

14 includes the step of indexing according to the identity 

15 of a team participating in the sports event. 

16 Additionally in accordance with a preferred 

17 embodiment of the present invention, the step of indexing 

18 includes the step of indexing according to the following 

19 group of action types: offense, and defense. 

20 Further in accordance with a preferred embodi- 

21 raent of the present invention, the step of indexing 

22 includes the steps of receiving at least one audio signal 

23 corresponding to at least one frame of the video se- 

24 quence, and keyword spotting the audio signal for indices 

25 in order to index the video sequence in accordance with 

26 detected indices. 



27 Further in accordance with .a preferred- embodi- 

28 ment of the present invention, the step of receiving 

29 includes the step of receiving the audio channel of the 

30 video sequence. 

31 Still further in accordance with a preferred 

32 embodiment of the present invention, the step of receiv- 

33 ing includes the step of receiving an audio message from 

34 a user indicating an index. 

35 Additionally in accordance with a preferred 

36 embodiment of the present invention, the foreground 
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1 object shape discriminator includes a ball recognizer 

2 operative to recognize a ball. 

3 Further in accordance with a preferred embodi- 

4 raent of the present invention, the discriminator includes 

5 a team uniform recognizer operative to recognize member 

6 of a sports team by at least one characteristic of his 

7 uniform. 

8 There is also provided, in accordance with 

9 another preferred embodiment of the present invention, a 

10 television sports event replay method including the steps 

11 of receiving a video sequence of at least a portion of a 

12 sports event featuring at least one object in action, 

13 selecting at least one of the objects in action, tracking 

14 the selected objects through the video sequence, and 

15 broadcasting a replay of the video sequence on television 

16 with the selected objects highlighted. 

17 There is also provided, in accordance with 

18 another preferred embodiment of the present invention, a 

19 television. sports event replay method including the steps 

20 of receiving a video sequence of a sports event segment, 

21 featuring at least one object in action, wherein first 

22 and second portions of the segment takes place in first 

23 and second portions of the playing field, respectively 

24 and the second and- first portions are not visible in the 

25 video representation of the first and second portions, 

26 respectively, of the sports event, and broadcasting on 

27 television a representation of the sports event segment 

28 in which the first and second portions of the playing 

29 field constantly appear. 

30 ' Further in accordance with a preferred embodi- 

31 ment of the present invention, the object highlighter is 

32 operative to draw the past trajectory of at least one 

33 tracked object. 

34 There is also provided, in accordance with 

35 another preferred embodiment of the present invention, a 

36 sports event video manipulating method for manipulating a 
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1 representation of a sports event, the method including 

2 grabbing and digitizing at least one video field, track- 

3 ing an object through a plurality of successive video 

4 fields, receiving input from the object tracker and 

5 highlighting the tracked object on each of the plurality 

6 of successive video fields, and converting output of the 

7 object highlighter into a video standard format. 

8 There is also provided, in accordance with 

9 another preferred embodiment of the present invention, a 

10 video imagery manipulating method for manipulating video 

11 imagery including grabbing at least one video field, 

12 tracking an object through a plurality of successive 

13 video fields, and receiving input from the object track- 

14 er and highlighting the tracked object on each of the 

15 plurality of successive video fields. 

16 There is also provided, in accordance with 

17 another preferred embodiment of the present invention, a 

18 sports event video manipulating method for manipulating a 

19 representation of a sports event, the method including 

20 grabbing and digitizing at least one video field, and 

21 receiving a plurality of digitized fields from the video 

22 field grabber, representing a corresponding plurality of 

23 small portions of an arena and composing a mosaic of 

24 fields representing a larger portion of the arena. 

25 There is also provided, in accordance with 

26 another preferred embodiment of the present invention, a 

27 sports event video manipulating method for manipulating a 

28 representation of a sports event, the method including 

29 grabbing and digitizing a video field sequence, and 

30 differentiating foreground objects in the video field 

31 sequence from background in the video field sequence, at 

32 least partly oh the basis of foreground object motion. 

33 There is also provided, in accordance with 

34 another preferred embodiment of the present invention, a 

35 sports event video manipulating method for manipulating a 

36 representation of a sports event, the method including 
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grabbing . and digitizing at least one video field, and 
differentiating foreground objects in the video field 
from background in the video field, at least partly on 
the basis of foreground object shape. 

There is also provided, in accordance with 
another preferred embodiment of the present invention, a 
sports event video manipulating method for manipulating a 
representation of a sports event, the method including 
grabbing and digitizing at least one video field, and 
differentiating a character-bearing foreground object in 
the video field from background in the video field, at. 
least partly on the basis of character recognition. 
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1 BRIEF DESCRIPTION OF THE DRAWINGS 

2 

3 The present invention will be understood and 

4 appreciated from the following detailed description, 

5 taken in conjunction with the drawings in which: 

6 Fig. 1 is a simplified block diagram of a 

7 sports event analysis system which is constructed and 

8 operative in accordance with a preferred embodiment of 

9 the present invention; 

10 Fig. 2 is a sample display of a video frame 

11 including a highlighted player and an indication of the 

12 past trajectory of the highlighted player; 

13 Figs. 3A and 3B, taken together, form a simpli- 

14 fied flowchart of a preferred method for highlighting a 

15 video representation of an object included in a video 

16 representation of a scene; 

17 Fig. 4 is a pictorial illustration of conver- 

18 sion of a sequence of narrow-field partially overlapping 

19 frames into a single image with a wide field of view; 

20 Figs. 5A and 5B, taken together, form a simpli- 

21 fied flow-chart of a preferred method for generating a 

22 global diagnostic image; and" 

23 Figs. 6A and 6B are two sample video frames 

24 illustrating two options for diagnosis offensive sets in 

25 a soccer game. 
26 

27 
28 
29 
30 
31 
32 
33 
34 
35 
36 
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DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS 



The role of video to analyze certain team games 



4 such as basketball, football and soccer, is rapidly 

5 growing. Using video allows both coaches and players to 

6 visualize what the players have done right, what they 

7 have done wrong and where they must improve. From a 

8 coaching standpoint it helps in preparing game strategies 

9 against different opponents. 

10 In addition to pre-garae analysis of opponent 

11 scouting tapes and post-game analysis for self improve- 

12 ment, there is also need for a fast editing and analysis 

13 tool for half-time situations to allow the coaches to 

14 make strategy adjustments during half time. 

15 There is also a growing need for sport commen- 

16 tator demonstration tools in TV studios. These are needed 

17 to educate the viewers, to show them the various strate- 

18 gies and typical moves and exercises of the game in order 

19 to increase their interest and ultimately to increase the 

20 ratings of the program and the station. 

21 Advertising .messages or sponsorship logos 

22 superimposed on such diagnostic screens or clips are very 

23 effective since they are usually displayed after an 

24 important tactic in the game. 

25 Before proceeding with a description of the 

26 apparatus depicted in Fig. 1, the following comments are 

27 pertinent: The uses of the apparatus .and method relate to 

28 video editing and include but are not limited to analysis 

29 of video clips captured during sports games. The appara- 

30 tus of this invention may include, or operate in conjunc- 

31 tion with, a general purpose video editing machine. 

32 The embodiments described below refer, but are 

33 not limited to digital video editing systems and can 

34 include, or operate in conjunction with analog video 

35 editors as well. 

36 Reference is now made to Fig. 1 which illus- 
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1 trates a sports event analysis system, referenced gener- 

2 ally 10 , including a video encoding/decoding unit 20, a 

3 host computer 30, a digital memory storing device 40, an 

4 image analyzer 50, an annotation aid such as a lightpen 

5 60 and a video monitor 70. 

6 The system 10 is operative to/perform editing, 

7 analysis and diagnostic representations of video segments 

8 of sports events. The video encoder/decoder unit 20, 

9 such as an Optibase JPEG-2000, is operative to grab and 

10 digitize a sequence of individual frames from the color 

11 video input, preferably in real - time (25 or 30 frames 

12 per second), and to convert a digital image back into 

13 analog video. 

14 This unit is preferably able to compress and 

15 decompress the video images so that longer video segments 

16 may be stored in the digital memory 40. The Optibase JPG- 

17 2000 board is using Motion JPEG algorithm for compres- 

18 sion; other algorithms, such as MPEG, may also be used. 

19 \ The video encoder/decoder receives the video 

20 input from a plurality of optional sources such as a 

21 Video Cassette player (VC) , a Video Disk player (VD) or 

22 . broadcasted transmission and also outputs live video. The 

23 frame grabber can grab and digitize a full frame or 

24 preferably, due to. camera scanning and players movements, 

25 to operate on single fields. 

26 Host computer 30 may comprise a pc 486DX-50 

27 with 8 MB RAM. The host computer preferably includes an 
.28 audio board, installed into one of the extension slots, 
29 that can store and synchronize at least one audio channel 
30. and may also include a user dependent or user independent 
3.1 key-word spotting unit. The computer preferably communi- 

32 cates with the following units: 

33 a. video monitor 70 which may comprise a JVC 21" 

34 RGB/YC/PAL; 

35 b. an- edit monitor 90 such as a SVGA 14 • ■ non-interlaced 

36 low radiation monitor; 
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1 c. one or more user input devices which may, for example, 



3 d. one or more user output devices which may, for exam- 

4 pie, include stereo speakers 100; 

5 e. digital memory storage device 40 which may comprise 2 

6 GigaBytes SCSI Hard Disk; and 

7 f . image analysis, tracking and registration unit 50, 

8 also termed herein "image analyzer 50", which is opera- 

9 tive to analyze and manipulate a sequence of digital 

10 images and to create diagnostic or processed output that 

11 can be displayed on the video monitor 70 and/or sent into 

12 the output video port. " 

13 The image analyzer 50 preferably comprises at 

14 least one dedicated electronic board that is installed in 

15 the host computer extension slots and communicates with 

16 its bus. The image analyzer 50 operates on the spread 

17 image or rather makes use of the compressed data associ- 

18 a ted with the spread image. 

19 The system of Fig. 1 is preferably operative to 

20 perform a plurality of types of analysis, manipulation 

21 and editing operations on a video representation of a 

22 sports event, such as but not limited to the following 

23 operations: 

24 a. Control of the video input device - the VD or 

25 VC input is remotely controlled from the computer screen. 

26 The operator may use a plurality of control functions 

27 such as: fast forward, fast search backwards, play (no 

28 compression), record 20 (go back 20 seconds, compress and 

29 store on disk) , stop. 

30 b. Indexing video segments - the operator can 

31 quickly mark start and end points of relevant video 

32 segments in PLAY mode. This is preferably done using 

33 means such as a light pen, touch screen or a trackball. 

34 It may also be implemented using the audio channel - the 

35 operator says key-words describing the video segment into 

36 the microphone and the system later uses an audio key- 



2 



include light pen 60, microphone 72 and a keyboard 80; 
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1 word spotting unit (for example, a user dependent speech 

2 recognition pc board) to locate the clip in the synchro- 

3 nized audio channel, 

4 The segments are indexed according to any 

5 suitable criteria, preferably user-selected criteria, 

6 such as but not limited to type of offensive exercise, 

7 defense type, identified player habits, and then logged 

8 into the memory. All segments having the same index can 

9 later be accumulated for editing, analysis and sorting 

10 out. 

11 c; : On-line graphical blending and audio dubbing - 

12 when recording a video segment into system memory the 

13 operator can add graphical overlays on the video for 

14 example by directly "writing" on the screen with a light 

15 pen or by adding a vocal description using the micro- 

16 phone. 

17 d. Editing and sorting indexed video segments - 

18 the operator may perform a plurality of editing opera- 

19 tions on each indexed segment such as but not limited to 

20 changing start/stop points, deleting clips and adding 

21 graphical overlays or animation clips. He may then create 
2 2 groups of segments to be later recorded on video cas- 

23 settes and distributed among the players. 

24 . e. Spotting. key-words on the original audio chan- 
2 5 nel of the video input such as players' names, to auto- 

26 matically accumulate the video segments associated with 

27 these players. This preferably utilizes a key-word 

28 spotting system; a speaker dependent key-word spotter 

29 demands a training phase before running each cassette. 

30 f. . Automatic tracking and ■ highlighting of objects 

31 selected on a first frame of a segment throughout the 

32 sequence of frames composing an event until the objects 

33 get out of the camera field of view as illustrated by way 

34 of example in Fig. 2. A past trajectory of the objects 

35 may be superimposed on the video as also depicted in Fig. 

36 2. 
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1 g. Creation of a wide field of view (FOV) back- 

2 ground image of the playing fields by mosaicking partial- 

3 ly overlapping narrow FOV consecutive frames, automatic 

4 placing of players and production of analysis video 

5 segments "played" on the global playing fields image as 

6 shown in Fig. 4. These video clips may later be edited 

7 and sorted as all others. 

8 A preferred method of employing the system of 

9 Fig. 1 to perform operation f is as follows: 

10 Each frame of the input video stream is 

11 grabbed, digitized and preferably compressed by the video 

12 encoder/decoder unit 20: Objects are marked on the first 

13 frame of the video stream by the system operator prefera- 

14 bly by using the light pen 60. The digital frame infor- 

15 mation is typically stored in memory 40 and can be 

16 accessed and spread by the decoder 20. The frame pixels 

17 data is then DMAed by means of a fast video bus into the 

18 image analyzer 50 that uses a real-time motion estimation 

19 chip such as SGS-THOMSON STI3220 and a dedicated circuit 

20 to estimate the motion and to track the marked objects of 

21 interest and to estimate the motion of background windows 

22 at video rate. ' . 

23 A graphical object enhancement overlay is then 

24 created and stored on the hard disk memory 40. When the 

25 operator, wishes to record the sequence on a VCR the 

26 frames are decoded in unit 20 and the corresponding 

27 graphical overlays are superimposed on each frame. The 

28 composed image is then analog converted and sent to the 

29 video output port. 

30 * A preferred method of employing the system of 

31 Fig. 1 to perform operation g is similar to that de- 

32 scribed for operation f except that the image analyzer 50 

33 is now used to perform registration of the current frame 

34 into the previous ones and to generate the global diag- 

35 nostic still image or video clip. The light pen 60 is 

36 used here also to manually "paint" the residual "holes" 
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1 



in the 



background global image in addition to marking 
of interest in the first frame of the replayed 



3 



2 



objects 
set. 



4 



Fig. 2 is a sample display of a video frame 



5 including a highlighted player and an indication of the 

6 past trajectory of the highlighted player. 



8 taken together, form a simplified flowchart of a pre- 

9 ferred method for highlighting a video representation of 

10 an object included in a sequence of video representa- 

11 tions of an event, such as a sport event, which may be 

12 performed by units 30 and 50 of Fig. 1. 

13 In step 100 the video encoder 20 grabs and 

14 digitizes the first video frame. 

15 In step 110, the system accepts a user's indi- 

16 cation of at least one object to be highlighted. For 

17 example, the user may indicate an object by touching its 

18 image on the first video frame of the sequence with light 

19 pen 60 of Fig. 1. 

20 In step 120, the video analyzer 50 of Fig. 1 

21 detects and identifies the boundaries of the selected 

22 objects, using a conventional edge detection method. For 

23 example,. Gaussian edge detection, as described in the 

24 following publication, has been found to be suitable: 

25 J. F. Canny, "A computational approach to 

26 edge detection", IEEE Trans. Pattern Analysis and Machine 

27 Intelligence, vol. 8, pp. 679-698, November 1986. 

28 The disclosure of the above-referenced publica- 

29 tion and of all publications cited therewithin is hereby 

30 incorporated by reference. 

31 Alternatively, however, other conventional edge 

32 detection methods may be employed such as those described 

33 in the following references: 

34 F.M. Dickley and K.M. Shanmugan, "An 

35 optimal frequency domain filter for edge detection in 

36 digital pictures", IEEE Transactions on Pattern Analysis 



7 



Reference is now made to Figs. 3A and 3B which, 
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1 and Machine Intelligence, PAMI-1 (1) :37-49, 1977. 

2 R.M. Haralick, "Digital step edge from 

3 zero-crossings of second directional derivatives ", IEEE 

4 Transactions on Pattern Analysis and Machine Intelli- 

5 gence, PAMI-6 (1) : 58-68, 1984. 

6 D. Marr and H. Hildreth, "Theory of edge 

7 detection", Proceedings of the Royal Society of London, 

8 B(207) :187-217, 1980. 

9 The disclosures of the above-referenced publi- 

10 cations and of all publications cited therewithin are 

11 hereby incorporated by reference. 

12 A problem that has been encountered and identi- 

13 fied as. a possible obstacle to the edge detection task is 

14 image degradation due to motion induced blur. This may be 

15 due either to camera scanning or to object motion. The 

16 motion blur identification technique such as the ones 

17 described in the following reference may be used as part 

18 of step 120 in Fig. 3A: 

19 R.C. Gonzalez and P. Wintz, "Digital 

20 image processing", Addi son-Wesley, 1977. 

21 In step 122 all edge pixels with an edge 

22 strength below a user-specified threshold are ignored. 

23 All those above are marked as candidate boundary pixels. 

24 Connected sequences of candidate boundary pixels 

25 ("strings") are then identified. A connected sequence is 

26 defined as the set of boundary pixels such that a path 

27 consisting solely of boundary pixels exists between any 

28 two pixels in the set. Strings which are very close to 

29 one another are connected. At this point, just one con- 

30 nected string normally exists for each marked object 

31 which is guaranteed to completely surround a "central 

32 region" of the object. 

33 In step 124, objects whose connected strip of 

34 edges was detected and defined in steps 120 and 122 are 

35 highlighted in the initial frame viewed by the user in 

36 the course of performing step 110. 
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1 In step 140, a frame index n is initialized to 

2 2, since forthcoming steps 150-230 are performed for each 

3 video frame from the second frame, i.e. the frame follow- 

4 ing the initial frame which the user employed in perform-. 

5 ing step 110, onward. 

6 In s tep 146, the video encoder 20 grabs and 

7 digitizes the n'th frame. 

8 In step 150, a tracking technique is employed 

9 to detect, in frame n, the objects marked in step 110. 

10 The boundaries of the marked objects detected on the 

11 first video frame of the session, or in the previous 

12 frame, are used to track the object throughout the 

13 succession of frames using the method of edge tracking. 

14 This is done by defining a dynamical search window cen- 

15 tered around the object location in the previous frame 

16 and performing the edge detection procedure inside this 

17 window. 

18 F rom the third frame on, the direction of the 

19 relative motion of the object may be predicted and the 

20 search window may then be centered on the predicted 

21 point. The first step in edge tracking applies one of the 
2 2 known edge detection algorithms mentioned above. 

23 The next steps are thresholding and continuity 

24 processes, as described above. Alternatively, the above 

25 tracking method may be replaced by the following method: 

26 when the object is large enough to embody well defined 

27 correlation windows or features, the. tracking may be 

28 executed by correlating such windows, taken from subse- 

29 quent frames. 

30 . For objects typically characterized by a bound- 

31 ing rectangle larger than 10x10 pixels, reliable feature 

32 detection is expected and the correlation tracking method 

33 usually has a better probability of success than the edge 

34 tracking technique. A combination of these .tracking 

35 methods and others may also be used. The tracking proce- 

36 dure takes into account the fact that there may be a 
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1 change of unification (zoom in and'out, and of objects' 

2 poses through the succession of flames 

3 in step 160, the system analyses ana detects 

4 the following situations: 

5 fusion". - two or more objects which have merged into 

, "splitting- - one "fused" object which has split into at 
ft least two separated objects; 

8 , occlusion - an object which is partially bio*- * 
another object ( player, referee or by the ball) ■ 

Two approaches may be implemented to identify 



10 
11 

12 these phenomena: 

13 a The internal region of a marked object is matched, by 
LL of a pixel correlation method, to the - J 

15 in the previous frame- When a large local mismatch is 
idenUfied it may be due to occlusion or fusion The 
17 ptxel correlation image should first be smoothed to 

r, r;:i objects „ scene ,p^.». 

20 L ball, are continuously detected and tracked from 
! frame to frame. If occlusion of a marked object, by one 
o the players or by the ball, occurs, fusion and split- 

23 ting may then be predicted. * method for distinguishing 

24 between a uniformly moving background and foreground 
ob ects moving in different velocities is described below 



26 with reference to steps 350-370 of Fig. 5A. 

in step 170. the system, after detection and 
28 localization of all tracked objects, executes • '""^ 
._.,..„,„» to monitor all objects that have 



x the camera field of view and those that are pre- 

31 dieted, according to their computed angular speed, to 

32 exit the field of view in the forthcoming frames. 

Along with the exiting monitoring, a reentry 

34 analysis is performed to identify those marked and previ- 

35 ously-tracked objects that have disappeared but have 
re tared the scene (i.e.. the camera field of view, in 
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1 the current frame. 

2 The probability of success of the reentry 

3 analysis is lower than the probability of success of 

4 disappearance analysis. 

5 The method used for the reentry analysis pref- 

6 erably includes the following steps: 

7 1) keeping an inventory of the global coordinates of 

8 exiting objects at the moment of their exit The computa- 

9 tion of the global coordinates of a given pixel of frame 
10 n is described with reference to steps 340-380 of Fig. 

« 5A "' 2) prediction of the location of the objects in 

13 global coordinates based on their last computed velocity 

14 vector. , 

15 3 ) conversion of the predicted object location from 

16 the global coordinates into reentry frame coordinates. 

17 In step 180, the program decides, according to 

18 a pre-defined criterion, if the number of marked objects 

19 in the field of view is sufficient to continue the proc- 

20 ess into the following frames. When the number of objects 

21 has decreased below a given threshold it becomes reasona- 

22 ble to assume that the set has degenerated into an insig- 

23 nificant scene and the program terminates. It is still 

24 under the operator's discretion to reactivate the code 

25 from the termination point on. 

26 Typically, in steps 190 and 200, the boundaries 

27 of the tracked objects in frame n are defined. Sometimes 

28 these boundaries have already been defined, e.g. in step 

29 150, when edge tracking has been used. The edge strength 

30 of the boundaries is thresholded according to a pre- 

31 defined criterion, connected sequences of candidate 

32 boundary pixels are identified, strings which are very 

33 close to one another are connected and the main connected 

34 string, guaranteed to enclose a central region, is iden- 

35 tified. , , 

36 In step 206 the past trajectories of the marked 
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2 

3 

4 

5 

6 

7 

8 

9 



X objects on the current frame are computed This P-ee 
involves computation of each object' s centroid (or any 
other reference point, in previous frames and converse 
into current frame coordinates through the global frame 

„f reference as explained in step 450 of Fig. SB. 
o£ reference. ^ ^ ^ mt have been 

, traced is used he rein to refer 

to any suitable emphasis of an individual object in a 
„f , nortion of an individual object, such 
" r Z< °n t imiterto. its boundary, as detected and 
defined in step 200, or to a manipulation of the object 
13 or object portion such as color change, shadowing, blinx- 
ing or adding an emphasizing element such as a framing 
e lent surrounding the object, an arrow continuous , y 
16 pointing at the object, a caption appropriate to the 
ob ect which travels continuously along therewith. Proper 
me Lures to prevent edge and ^ture ; Ua S ,, » * 

1, when applying any part of these dynamic emphasizing 

2 . tecnnisues.^ ^ ^ trajectorles of 
2 2 the marked objects may also be superimposed on the video 
„ frame in step 2«. ^ ^ localizatlon of objects of 
25 interest in the global coordinate system, required for 
it the reentry analysis is executed. 

27 L step 230 the frame number is advanced and 

28 the highlighting process is repeated. analysis 

A problem usually encountered in the analysis 
of team games is the difficulty in conceptualizing a 
whole wide field of view of an offensive or 
tactic out of the succession of partially overlapping 



30 

31 whole wide 

" vt^f rames "tnat 'were captured by at least one TV camera 

34 ustn, relatively narrow fields of view centering around 

35 

36 ers 



using relatively narrow — 

the instantaneous location of the ball and active play- 
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! The system of the present invention is pref era- 

2 bly characterized in that one large pseudo-real image of 

3 the playing fields, or a relatively large portion of 

4 thereof, is mosaiced from among the successive partially 

5 overlapping video frames generated by the camera in the 

6 course of the event. ^.u^- 

7 The frames are first composed in mosaic fashion 

8 to create a global background image onto which the active 

9 players and the ball are placed in their accurate loca- 

10 tions using their real images or graphical icons thus 

11 demonstrating a comprehensive representation of all the 

12 moves composing the event and allowing better understand- 

13 ing of the event. 

Fig . 4 is a pictorial illustration of conver- 

15 sion of a sequence of narrow-field partially overlapping 

16 frames into a single image with a wide field of view 

17 Reference is now made to Figs. 5A and 5B which, 

18 taken together, form a simplified flowchart of a pre- 

19 ferred method for generating this global diagnostic image 

20 which may be performed by the image analyzer 50 and host 

21 computer 30 of Fig. 1. 

22 In optional step. 310, the operator manually 

23 selects the objects of interest whose performance he 

24 wants to analyze through the replayed set. This is done 

25 using an annotation aid such as a light pen or a mouse. 

26 When this step is omitted, the system preferably consid- 

27 ers all subsequently identified foreground objects m the 
28- image as objects of interest. 

29 it is also possible that the system will auto- 

30 matically classify detected foreground objects according 

31 to their colors, identified back numbers of players, 

32 geometrical shape (for example, the round shape of a 

33 ball) and/or other attributes and will select a subset of 

34 foreground objects to be defined as objects of ^erest 

35 Although the main implementation of this method 

36 will be during replay situations, it is possible to 
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1 employ the method in the real time of the event. In such 

2 cases the automatic selection of objects of interest is 

3 normally a necessity, 
in step 320, the first frame or first field xs 

grabbed by the video encoder 20, digitized and, optional- 
grabbed y ^ glQbal . mage< 

lv compressed, to serve ao 

in step 330, a frame index n is initialized to 



in step ' , 

8 2 since forthcoming steps 340-450 are performed for each 
video frame from the second frame, i.e. the frame c , o„- 



9 
10 



video iranie j.j-«"> 

ing the initial frame which the user employed in perform- 

11 ing step 310, onward. 
' in step 340, the nth video frame is grabbed, 

13 digitized and, optionally, compressed by the unit 20 
.I in step 350, the area of the frame is divided 



20 
21 
22 
23 



ill — ' 

15 into a large number, typically 100, of separate contx*.- 
" ous small window areas. The windows are then classified 

17 and each of them is given a weight represents its 

..„,. in step 360, the correla- 

18 information content. in f 

19 tion value (or another value such as the sum o ab olute 
differences) of each window of frame n with the corre 
spending window in the (n-1) • th frame for each point 
inside a given search window, is computed. 

The matrix of correlation values is then fitted 

24 to a 2-d polynomial to achieve the minimum value with 

25 sub-pixel accuracy. The- vector motion associated with 

26 each correlation window is then computed from the meas- 

27 ured shift and frame periods. 

The average value, considered the "majority 

29 motion" velocity, is then computed. The deviating . windows 

30 are excluded and the global motion is remeasured. This 

31 process is iteratively continued until convergence is 

32 achieved. In each iteration, smaller window si.es may be 

33 used to obtain a better spatial resolution. 

] 4 In step 370. the local window shift information 

35 is used to distinguish between foreground objects, i.e 

36 moving objects, and background regions in the image which 
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x have the --iority motion" velocity that 1. usually due 

? M MBera 3SC the identified "background re- 

■ . of frame n are registered into the (■»-!>•«. frame. 

4 T this " Lne successively o. all fra.es composing 

5 Srnce this is dene backg round 

6 ^"''iLrelent ll" loosed in mosaic fashion fro. 
' farrow Z l rttally overlapping fra.es. The varping 

cess H executed using proper averaging since contri- 
process i • «~» in the global ina ,e may come 

rr:re Vn ^ ^ » «. — ^ 



. 8 
9 
10 

11 



Xi successive frames^ ^ ^ . subset of 



«-3 In step j^w, ■ -.. - . , ^x. 

nMprts in the current frame that 

•^ fln <-ifipd foreground objects i« >-" c 

identified rorey ^ -- „ 1ont . pf , either automatically or 

20 Strl ° 9 :° r t " interest are then computed in the local 
•>■»■ is stored for later use. 

" m step <i0. an "end of session" decision is 

,3 taken by the system. The criterion for such a decision 

25 taicen me number< a 

26 ^^iTZTl^Tlll -relation mechanism. 
11 ma:or failure of inltla ted by the TV produc- 
11 T'Tr^Z in which the number of identified 

k'.' L of interest in the camera FOV has decreased to a 

30 ob f CtS Of J; nte a r ; S re . prograra „ e a threshold. When -end of 

31 rion^rsVclLed^he program stops the fie- of video 

a , n nHnues with Steps 430-460 of Fig. 5B. 
33 frames and on t ues^ ^ ^ £ _ e 

36 frame. 
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x in step 430, the system performs an automatic 

2 "hole filling" process on the global- image mosaic. As 
described above, each pixel in each frame is classified 
as a background or foreground pixel. If it is a back- 

5 ground pixel, it is used to update the corresponding 

6 oixel of the global image when added with the proper 

7 weighting, if the pixel is classified as a "foreground 

8 Pixel", its value is ignored. The probability that a 

9 given background pixel of the global image will be oc- 
cluded with foreground objects through the whole succes- 

xx sion of overlapping frames is typically low but non-zero 

12 and the existence of consequent "holes" in the global 

13 image cannot, therefore, be ruled out. 

14 . m step 430 these holes are automatically 

15 "filled" using "spatial averages of neighboring pixel 



10 

11 



16 values. 
17 



In step 440, the user optionally paints any 

18 residual holes which may remain after the execution of 

19 automatic hole painting step 430. 

20 This may be effected using annotation aid 60 

21 and standard graphics software. At this point, the user 

22 may also "eliminate" residual foreground objects that 

23 have not been discriminated by the automatic process, 

24 such as stationary, players. In step 450, the 

25 system registers the identified objects of interest 

26 whose global coordinates have been computed for each 

27 frame in step 390, on the global background image at 

28 various time points. The system may automatically display 

29 all the objects or enable the operator to select, only a 

30 few objects to be displayed on the diagnostic screen. 

31 - In step 460, the system displays the diagnostic 

32 image or video clip on the video monitor 70, the computer 

33 monitor 90, and/or sorts it into the video output port of 

34 unit 20 for recording or transmission. At .least the 

35 following display options of the diagnosis information 

36 are preferably provided: 



c 



„,„ PCT/US94/11S27 
WO 95/10915 



27 

1 a. icons of the objects of interest, typically including 

2 players and a ball, are superimposed on the global 

3 background image at each selected point of time and the 

4 temporal evolution of the set is displayed in a film-like 

5 manner.. 

6 b. The images of the objects of interest themselves are 

7 superimposed and the time evolution is displayed as a 

8 video clip resembling the original video output • except 

9 for the two following differences: 
1. the background is a wide angle view embodying 



10 

11 a substantial portion of the playing fields thereby to 

12 orient a spectator in the field and to afford the specta- 

13 tor a better understanding of the tactical moves. 

14 2 . players that did not take part in the set are 

15 "eliminated" so that the spectators' attention is focused 

16 on the active players. 

17 c. Trajectories of players and/or ball may also be auto- 

18 matically displayed with optional time marking points. 

19 d. Still images summarizing the entire event by superim- 

20 posing acts that occurred at different time points on the 

21 same global background image. 

2 2 Reference is now made to Figs. 6A and 6B that 

23 represent two sample video frames illustrating two op- 

24 tional display formats of diagnostic still images in a 

25 soccer game pertaining to option d. above. It is 

26 appreciated that various features of the invention which 

27 are, for clarity, described in the contexts of separate 

28 embodiments may also be provided in combination in a 

29 single embodiment. Conversely, various features of the 

30 invention which are, for brevity, described in the con- 

31 text of a single embodiment may also be provided- sepa- 

32 rately or in any suitable subcombination. 

33 : In a n embodiments shown and described hereina- 

34 bove, either frames or individual fields may be raanipu- 

35 lated, according to alternative embodiments of the inven- 

36 tion.' Preferably, individual fields are manipulated. 



f 



X 



PCT/US94/11527 

WO 95/10915 



28 

in 



1 it will be appreciated by persons skilled 

2 the art that the present invention is not limited to what 

3 has been particularly shown and described hereinabove. 

4 Rather, the scope of the present invention is defined 

5 only by the claims that follow: 

6 
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13 
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1 
2 

3 CLAIMS 



9 
10 
11 



13 
14 
15 



4 

s 1 A sports event video manipulating system for 

6 manipulating a representation of a sports event, the 

7 sports editor comprising: 

8 a video field grabber operative to grab at 
least one video field including a video image A/D con- 
verter operative to digitize a grabbed video field; 

an object tracker operative to track an object 
12 through a plurality of successive video fields; 

an object highlighter receiving input from the 
object tracker and operative to highlight the tracked 
object on each of the plurality of successive video 

16 fields; 

17 a D/A image converter operative to convert 

18 output of the object highlighter into a video standard 

19 format; and 

20 a video display monitor. 

21 -A 

2 2 2 . A "system according to claim 1 wherein said 

23 video field grabber is replaced by a video frame grabber. 
24 

25 3 e A system according to claim 1 and also compns- 

26 ing a marking device allowing a user to indicate an 

27 object of interest and providing an output indication of 
the object of interest to the object tracker. 



28 

29 
30 



4. -A system according to claim 1 and also compris- 

31 ing a video editor operatively associated with the 

32 video field grabber and with the video display monitor. 



33 
34 



_ 5 . a system according to claim 1 wherein said 

35 object tracker includes an object exiting monitor opera- 



36 



tive to monitor for exit of an object from at least 



one 
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video field. 



5 . A system according to claim 1 wherein said 

object tracker is operative to track a plurality of 
objects. 

7 A system according to claim 6 wherein said 

object tracker includes an occlusion predictor operative 
to predict occlusion of at least one of the plurality of 
tracked objects by at least one other object. 

8> A system according to claim 6 wherein said 

object .tracker comprises a moving object identifier 
operative to identify objects which are in motion at 
least some of the time "as the plurality of objects to 
tracked. 



be 



9< A system according to claim 8 wherein said 

moving object identifier comprises a color analyzer 
operative to distinguish at least one color characteriz- 
ing objects at least sometimes in motion from at least 
one background color characterizing stationary objects. 

10. A system according to claim 1 wherein said 

object tracker comprises actual location generating means 
operative to generate and store an indication of an 
actual location of a tracked object at an individual 
time. 

H. A system according to claim 1 wherein said 

object tracker comprises f ield-of-view determining means 
operative to provide an indication of the actual direc- 
tion of a current field of view center. 



12. 



A video imagery manipulating system for manipu- 
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1 lating video imagery comprising: 

2 a video field grabber operative to grab at 

3 least one video field; 

4 an object tracker operative to track an object 

5 through a plurality of successive video fields; and 
an object highlighter receiving input from the 

object tracker and operative to highlight the -tracked 
object on each of the plurality of successive video 
9 fields. 
10 

X1 13> A sports event video manipulating system for 

12 manipulating a representation of a sports event, the 

13 sports editor comprising: 

14 a video field grabber operative to grab at 
15 least one video field including a video image A/D con- 

16 verter operative to digitize a grabbed video field; and 

17 a field mosaic composer operative to receive a 

18 plurality of digitized fields from the video field grab- 

19 ber, : representing a corresponding plurality of small 
portions of an arena and to compose a mosaic of fields 



20 



21 representing a larger portion of the arena, 



22 
23 



14> a system according to claim 13 and also com- 

24 prising a field mosaic graphic output system operative 

25 to provide a visually sensible representation of the 

26 larger portion of the arena. 



27 
28 



15. A system according to claim 14 wherein the 

29 field mosaic graphic output system comprises: 

30 - a video display; and 

31 - . a D/A converter operative to convert a digital 

32 representation of the field mosaic to a video representa- 

33 tion thereof. 
34 

35 16 . A system according to claim 14 wherein the 

36 field mosaic graphic output system comprises a computer 
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1 screen, 

4 

5 
6 



. ^ system a 

; field grabber comprises a video frame grabber 
6 vl deo f^ld grab* ^ wherein 

eo 'frame ana herein the «.» 
10 wise, a frame mosaic composer operative to » eel ™ 

P p r Uty of digitized frames from the videc , rame grah 
12 Lr and to compose therefrom a mosaic of frames. 

" ' A system according to claim 13 "herein the 

18 the distances themselves. 

20 a system according to claim 13 wherein the 

" Plurality of small portions are viewed from a smgle 

22 location. 

" A sports event video manipulating system for 

manipulating a representation of a sports event. 



A sports event- - 

24 21. &F . e „ cnnrfs event, the 

25 

9fi snorts editor comprising: 
" a video field grabber operative to grab a video 

27 ... a video image A/D converter 

28 

29 

. , m ^ion-based foregrounu-u-^y- — 

in 
video 



„„ a -inrludinq a video image A/D cc 

op«- — s ; fcegrom „ a discrimi. 

aM H ve to differentiate foreground objects n 
r 2 r -d se q ue„ce from hac.ground in the vide 

33 field se,uence. at least partly on the hasis of fore- 

34 ground object motion. 
A system according to claim 21 also comprising 



35 

36 22 
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1 a field mosaic composer operative to receive a sequence 

2 of pluralities of digitized fields from the video field 

3 grabber, each plurality of digitized fields representing 

4 a corresponding plurality of small portions of an arena 

5 and to compose a sequence of mosaics of fields, each 

6 mosaic representing a larger portion of the arena, 

7 and wherein said discriminator is operative to 

8 differentiate foreground objects in the sequence of 

9 mosaics. 

^ 23 A sports event video manipulating system for 

12 manipulating a representation of a sports event, the 

13 sports editor comprising: 

14 a video field grabber operative to grab at 

15 least one video field including a video image A/D con- 

16 verter operative to digitize a grabbed video field; and 

17 a foreground object shape foreground-background 

18 discriminator operative to differentiate foreground 

19 objects in the video field from background in the video 

20 field, at least partly on the basis of foreground object 

21 shape. 
22 

23 24. A sports event video manipulating system for 

24 manipulating a representation of a sports event, the 

25 sports editor comprising: 

26 a video field grabber operative to grab at 

27 least one video field including a video image A/D con- 

28 verter operative to digitize a grabbed video field; and 

29 a foreground object shape foreground-background 

30 discriminator operative to differentiate a character- 

31 bearing foreground object in the video field from back- 

32 ground in the video field, at least partly on the basis 

33 of character recognition. 

35 25. A system according to claim 21 wherein said 

36 discriminator differentiates foreground objects from 
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background objects at least partly on the basis of color. 



! , 6 A system according to claim 21 and also com- 

4 prising a foreground object eliminator operative to 
4 prising a * . reP i a ce them with 

eliminate foreground objects and repiac 



by&tcm ws^w^ — w 

foreground object eliminator operative to 
foreground objects 
adjacent background information. 

o 27 A system according to claim 13 and also com- 
prising a selected object replacer operative to identify 
pri&Auy o „„i Q „4- 0 rl nbiect with 



9 
10 

11 an icon in the mosaic 



r:Zted object and to replace the selected object with 



" 28 . .a video indexing method comprising the steps 

•V providing a digital representation of a video 

16 sequence featuring at least one object performing at 

17 least one type of action; and 
18 

19 least one index. 

Z 29 A method according to claim 28 wherein the step 

22 of' indexing comprises the step of indexing according to 



type ox cit; uv«/ 

indexing the video sequence according to at 



23 the identity of the object. 

M 30 A method according to claim 28 wherein the step 

26 of' indexing comprises the step of indexing by action 

27 type. 

" 31 A method according to claim 29 wherein the 

30 video sequence represents a sports event and the step of 

,1 Indexing according to the identity of the object com- 

3 prises the step of indexing according to the 

33 at least one player participating in at least a port.on 

34 of the sports event. 

3e 32 . a method according to claim 31 wherein the step 
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of indexing comprises the step of indexing according to 
the identity of a team participating in the sports event. 



1 
2 

3 A method according to claim 30 wherein the step 



4 33 
5 



12 
13 
14 



of' indexing comprises the step of indexing according to 

6 the following group of action types: 

7 offense; and 
6 defense. 

g 

10 . 34. A video sorting method according to claim 28 

11 wherein the step of indexing comprises the step of: 
receiving at least one audio signal correspond- 
ing to at least one frame of the video sequence; and 

keyword spotting the audio signal for indices 

15 in order to index the video sequence in accordance with 

16 detected indices. 

\l 35 A method according to claim 34 wherein the step 

19 of receiving comprises the step of receiving the audio 

20 channel of the video sequence. 

22 36 A method according to claim 34 wherein the step 

23 of receiving comprises the step of receiving an audio 

24 message from a user indicating an index. 

" 37. A system according to claim 23 wherein the 

27 foreground object shape discriminator comprises a ball 

28 recognizer operative to recognize a ball. 

29 . .. 

30 38 A system according to claim 25 wherein said 

31 discriminator comprises a team uniform recognizer opera- 

32 tive to recognize member of a sports team by at least one 

33 characteristic of his uniform. 

35 39 . A television sports event replay method com- 

36 prising the steps of: 
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! receiving a video sequence of at ieast a por- 

2 tion of a sports event featuring at least one object in 

3 action; 

4 selecting at least one of the objects in ac- 

5 tion; 

6 tracking the selected objects through the video 

7 sequence; and 

8 broadcasting a replay of the video sequence on 

9 television with the selected objects highlighted. 
10 

H 40. A television sports event replay method cora- 

12 prising the steps of: 

13 receiving a video sequence of a sports event 

14 segment,' featuring at least one object in. action, wherein 

15 first and second portions of the segment takes place in 

16 first and second portions of the playing field, respec- 

17 tively and the second and first portions are not visible 

18 in the video representation of the first and second 

19 portions, respectively, of the sports event; and 

20 broadcasting on television a representation of 

21 the sports event segment in which the first and second 

22 portions of the playing field constantly appear. 

24 41. A system according to claim 1 wherein said 

25 object highlighter is operative to draw the past trajec- 
tory of at least one tracked object. 



26 
27 

28 42 



A sports event video manipulating method for 

29 manipulating a representation of a sports event, the 

30 method comprising: 

31 grabbing and digitizing at least one video 

32 field; 

33 tracking an object through a plurality of 

34 successive video fields; 

35 receiving input from the object tracker and 

36 highlighting the tracked object on each of the plurality 
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1 of successive video fields; and 

2 converting output of the object highlighter 

3 into a video standard format. 

5 43. A video imagery manipulating method for manipu- 

6 lating video imagery comprising: 
grabbing at least one video field; 
tracking an object through a plurality of 

9 successive video fields; and 

10 receiving input from the object tracker and 

11 highlighting the tracked object on each of the plurality 

12 of successive video fields. 

[] 44 "a sports event video manipulating method for 

15 manipulating a representation of a sports event, the 

16 method comprising: 

17 grabbing and digitizing at least one video 

18 field; and 

19 receiving a plurality' of digitized fields from 

20 the video field grabber, representing a corresponding 

21 plurality of small portions of an arena and composing a 

22 mosaic of fields representing a larger portion of the 

23 arena. 
24 

25 45 A sports event video manipulating method for 

26 manipulating a representation of a sports event, the 

27 method comprising: 

28 grabbing and digitizing a video field sequence; 

29 and • 4-k 

30 differentiating foreground objects m the 

31 video field sequence from background in the video field 
32 sequence, at. least partly on the basis of foreground 

33 object motion. 

3 5 46 A sports event video manipulating method for 

36 manipulating a representation of a sports event, the 
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1 method comprising: 

2 grabbing and digitizing at least one video 

3 field; and 



13 
14 
15 



differentiating foreground objects in the video 

5 field from background in the video field, at least partly 

6 on the basis of foreground object shape. 

8 47 a sports event video manipulating method for 

9 manipulating a representation of a sports event, the 
10 method comprising: 

H grabbing and digitizing at least one video 

12 field; and 

differentiating a character-bearing foreground 
object in the video field from background in the video 
field, at least partly on the basis of character recogm- 

16 tion. 

17 

18 

19 

20 . : • 

21 

22 

23 . 

24 . 
25 
26 
27 
28 

' 29 
30 
31 
32 
33 
34 
35 
36 
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